Bayesian Inference on Principal Component Analysis Using Reversible Jump Markov Chain Monte Carlo

Based on the probabilistic reformulation of principal component analysis (PCA), we consider the problem of determining the number of principal components as a model selection problem. We present a hierarchical model for probabilistic PCA and construct a Bayesian inference method for this model using reversible jump Markov chain Monte Carlo (MCMC). By regarding each principal component as a point in a one-dimensional space and employing only birthdeath moves in our reversible jump methodology, our proposed method is simple and capable of automatically determining the number of principal components and estimating the parameters simultaneously under the same disciplined framework. Simulation experiments are performed to demonstrate the effectiveness of our MCMC method. Introduction Principal component analysis (PCA) is a powerful tool for data analysis. It has been widely used for such tasks as dimensionality reduction, data compression and visualization. The original derivation of PCA is based on a standardized linear projection that maximizes the variance in the projected space. Recently, Tipping & Bishop (1999) proposed the probabilistic PCA which explores the relationship between PCA and factor analysis of generative latent variable models. This opens the door to various Bayesian treatments of PCA. In particular, Bayesian inference can now be employed to solve the central problem of determining the number of principal components that should be retained. Bishop (1999a; 1999b) addressed this by using automatic relevance determination (ARD) (Neal 1996) and Bayesian variational methods. Minka (2001), on the other hand, adopted a Bayesian method which is based on the Laplace approximation. In this paper, we propose a hierarchical model for Bayesian inference on PCA using the novel reversible jump Markov chain Monte Carlo (MCMC) algorithm of Green (1995). In brief, reversible jump MCMC is a random-sweep Metropolis-Hastings method for varying-dimension probCopyright c © 2004, American Association for Artificial Intelligence (www.aaai.org). All rights reserved. lems. It constructs a dimension matching transform using the reversible jump methodology and estimates the parameters using Gibbs sampling. Richardson & Green (1997), by developing the split-merge and birth-death moves for the reversible jump methodology, performed a fully Bayesian analysis on univariate data generated from a finite Gaussian mixture (GM) with an unknown number of components. This was then further extended to univariate hidden Markov models (HMM) by Robert, Rydén, & Titterington (2000). In general, reversible jump MCMC is attractive in that it can perform parameter estimation and model selection simultaneously within the same framework. In contrast, the other methods mentioned above can only perform model selection separately. In recent years, reversible jump MCMC has also been successfully applied to neural networks (Holmes & Mallick 1998; Andrieu, Djurié, & Doucet 2001) and pattern recognition (Roberts, Holmes, & Denison 2001). Motivated by these successes, in this paper, we introduce reversible jump MCMC into the probabilistic PCA framework. This provides a disciplined method to perform parameter estimation simultaneously with choosing the number of principal components. In particular, we propose a hierarchical model for probabilistic PCA, together with a Bayesian inference procedure for this model using reversible jump MCMC. Note that PCA is considerably simpler than GMs and HMMs in the following ways. First, PCA has much fewer free parameters than GMs and HMMs. Second, unlike GMs and HMMs, no component in PCA can be empty. Third, using reversible jump MCMC in GMs and HMMs for high-dimensional data is still an open problem, while reversible jump MCMC for PCA is more manageable because, as to be discussed in more detail in later sections, each principal component can be regarded as a point in some onedimensional space. Because of these, we will only employ birth-death moves for the dimension matching transform in our reversible jump methodology. The rest of this paper is organized as follows. In the next section, we give a brief overview of probabilistic PCA and the corresponding maximum likelihood estimation problem. A hierarchical Bayesian model and the corresponding reversible jump MCMC procedure are then presented, followed by some experimental results on different data sets. The last section gives some concluding remarks. Probabilistic PCA Probabilistic PCA was proposed by Tipping & Bishop (1999). In this model, a high-dimensional random vector x is expressed as a linear combination of basis vectors (hj’s) plus noise ( ):